Measuring the Impacts of Teachers: Reply to Rothstein∗

Using data from North Carolina, Jesse Rothstein (2017) presents a comprehensive replication of Chetty, Friedman, and Rockoff’s [CFR] (2014a,b) results on teachers’ impacts. In addition, Rothstein presents new evidence that he argues raises concerns about three aspects of CFR’s methods and identification assumptions: their treatment of missing data, the validity of their quasi-experimental design, and their method of controlling for observables when estimating teachers’ long-term effects. In this paper, we show that Rothstein’s methodological critiques are not valid by replicating his new empirical findings using simulated data in which none of CFR’s identification assumptions are violated. We also present supplementary empirical evidence from our data supporting the assumptions required for CFR’s original analyses. Together, these results show that: (1) Rothstein’s technique for imputing teacher VA for teachers with missing data generates bias, while subsamples with no missing data yield estimates of forecast bias similar to CFR’s baseline results; (2) his proposed prior score “placebo test” rejects valid quasiexperimental research designs, and the correlation between changes in prior test scores and current teacher value-added he documents is an artifact of estimating teacher value-added using prior test score data; and (3) his method of controlling for covariates yields inconsistent estimates of teachers’ long-term effects, while quasi-experimental designs that do not rely on controls for observables yield estimates of teachers’ long-term impacts similar to CFR’s baseline results. We conclude that Rothstein’s important replication study is entirely consistent with – and in fact reinforces – CFR’s methods and results. Our conclusions match those of Bacher-Hicks, Kane, and Staiger (2014), who replicate both CFR’s results and Rothstein’s findings using data from Los Angeles and also conclude that Rothstein’s results raise no concerns about CFR’s analysis. ∗We thank Gary Chamberlain, Lawrence Katz, and Michael Stepner for helpful comments and Augustin Bergeron, Nikolaus Hildebrand, and Benjamin Scuderi for research assistance. On May 4, 2012, Chetty was retained as an expert witness by Gibson, Dunn, and Crutcher LLP to testify about the importance of teacher effectiveness for student learning in Vergara v. California, a case that was decided on June 10, 2014, before research on this paper began. On June 13, 2015, Friedman was retained as a potential expert witness by the Radey Law Firm to advise the Florida Department of Education on the importance of teacher effectiveness for student learning, as related the development of Draft Rule 6A-5.0411. His work on this case lasted for approximately one week and he had no involvement in any legal proceeding. On December 11, 2015, Friedman was retained as an expert witness by the Houston Independent School District in the matter of Houston Federation of Teachers, et al. v. Houston Independent School District. Friedman has not shared or discussed this paper with the parties in that case or their attorneys. In two recent papers, Chetty, Friedman, and Rockoff (2014a,b) [CFR] measure teachers’ effects on students’ test scores and long-term outcomes. The first paper [hereafter, CFR-I] measures the degree of bias in teacher value-added (VA) estimates using a research design that exploits changes in teaching staff across years within schools, regressing changes in mean test scores across cohorts on changes in mean teacher VA. The second paper [hereafter, CFR-II] measures teachers’ long-term impacts on students’ earnings and other outcomes. CFR’s main conclusions are that (1) teacher VA estimates exhibit little “forecast bias” – that is, they provide accurate predictions of teachers’ causal effects on student achievement on average – and (2) teachers with high test-score VA also improve their students’ long-term outcomes, such as college attendance rates and earnings. Recent studies have replicated CFR’s findings, which were based on data from New York City, using data from other school districts.1 Jesse Rothstein (2016) successfully replicates CFR-I’s results on bias in teacher VA using data from North Carolina and presents evidence on teachers’ long-term impacts on students’ outcomes in high school that is consistent with CFR-II’s findings. Andrew Bacher-Hicks, Thomas Kane, and Douglas Staiger (2016) [hereafter BKS] replicate CFRI’s results using data from the Los Angeles Unified School District. These studies collectively demonstrate that value-added measures of teacher quality exhibit very consistent properties; as Rothstein (2016) puts it, “the empirical results are remarkably robust across disparate settings.” Such replications are extremely valuable for evaluating the robustness of CFR’s results and we are indebted to Rothstein and Bacher-Hicks, Kane, and Staiger for conducting these studies. In addition to replicating CFR’s findings, Rothstein (2016) raises three concerns about the validity of CFR’s methods: the treatment of missing data, the validity of the quasi-experimental design, and the method of controlling for observables when estimating teachers’ long-term effects. We are grateful that Jesse Rothstein has invested great effort in re-examining CFR’s methodology, especially since our own interest in teacher value-added was sparked in part by Rothstein’s (2010) paper on this subject. We have learned a great deal from his work on this topic. However, we respectfully disagree with each of the points he makes. We show that Rothstein’s methodological critiques are not valid by replicating each of his new empirical findings using simulated data in which none of CFR’s identification assumptions are violated. We therefore conclude that Rothstein’s findings are entirely consistent with CFR’s methods and results. Our conclusions match those of BKS (2016), who also investigate the issues Rothstein raises in their data and conclude that they CFR-I and CFR-II did not name the school district, but it was identified in the Vergara v. California trial.